1. Field of the Invention
The present invention relates to load balancing for networks with bundles of communication links between one network device and at least one pair of different network devices, such as a pair of provider edge nodes through which customer networks are connected to a provider network, such as a metropolitan area network (MAN).
2. Description of the Related Art
Networks of general purpose computer systems and special purpose devices connected by external communication links are well known and widely used in commerce. The networks often include one or more network devices that facilitate the passage of information between the computer systems and special purpose devices. A network node is a network device or computer system or other special purpose device connected by the communication links.
Information is exchanged between network nodes according to one or more of many well known, new or still developing protocols. In this context, a protocol consists of a set of rules defining how the nodes interact with each other based on information sent over the communication links. The protocols are effective at different layers of operation within each node, from generating and receiving physical signals of various types, to selecting a link for transferring those signals, to the format of information indicated by those signals, to identifying which software application executing on a computer system sends or receives the information. The conceptually different layers of protocols for exchanging information over a network are described in the Open Systems Interconnection (OSI) Reference Model. The OSI Reference Model is generally described in more detail in Section 1.1 of the reference book entitled Interconnections Second Edition, by Radia Perlman, published September 1999, which is hereby incorporated by reference as though fully set forth herein.
Communications between nodes are typically effected by exchanging discrete packets of data. Each packet typically comprises 1] header information associated with a particular protocol, and 2] payload information that follows the header information and contains information to be processed, often independently of that particular protocol. In some protocols, the packet includes 3] trailer information following the payload and indicating the end of the payload information. The header includes information such as the source of the packet, its destination, the length of the payload, and other properties used by the protocol. Often, the data in the payload for the particular protocol includes a header and payload for a different protocol associated with a different, usually higher layer of the OSI Reference Model. The protocol in the payload is said to be encapsulated in the protocol of the header. The headers included in a packet traversing multiple heterogeneous networks, such as the Internet, typically include a physical (layer 1) header, a data-link (layer 2) header, an internetwork (layer 3) header and a transport (layer 4) header, as defined by the Open Systems Interconnection (OSI) Reference Model. The physical layer protocol identifies the physical media used to communicate with another device. The data-link protocol identifies a different node connected by the physical media to a particular node.
Some protocols span the layers of the OSI Reference Model. For example, the Ethernet local area network (LAN) protocol includes both layer 1 and layer 2 information. The International Electrical and Electronics Engineers (IEEE) 802.3 protocol, an implementation of the Ethernet protocol, includes layer 1 information and some layer 2 information.
Routers and switches are network devices that determine which communication link or links to employ to support the progress of packets through the network. For example, Ethernet switches forward packets according to the Ethernet protocol at layer 2 based on unique identifiers for each device connected. Some current routers implement sophisticated algorithms that provide high performance forwarding of packets based on combining layer 2 and layer 3 header information, or some other combination, and knowledge of network topology beyond the adjacent nodes.
As used here, a port is a physical interface on a network device that is, or can be, connected to a cable to serve as a communication link with a port on another network device. For example, three types of widely used Ethernet ports have three different bandwidths of 100 Megabits per second (Mbps, where 1 Megabit=106 binary digits called bits), 1 Gigabit per second (Gbps, where 1 Gigabit=109 bits), or 10 Gbps. These three ports and links are termed Fast Ethernet, Gigabit Ethernet and 10 Gigabit Ethernet, respectively. Optical metropolitan area networks (MANs) and wide area networks (WANs) have been developed using special optical ports on nodes that repeat and switch optical signals, correcting for attenuation and other signal degradation. Optical protocols for these networks include a Synchronous Optical Network (SONET), Synchronous Digital Hierarchy (SDH) and Optical Transport Network (OTN) protocols.
In some circumstances, such as for redundancy or to provide needed bandwidth, some networks bundle multiple communication links between a pair of network nodes. For example, if network traffic between a particular server and an Ethernet switch in an office building needs bandwidth up to 500 Mbps, then it might be more cost-effective to connect five Fast Ethernet ports on each device rather than to install a Gigabit Ethernet port on each device and string a single Gigabit cable in the walls between them. Also the five Fast Ethernet links provide device resiliency against hardware failure on one port or link. The five Fast Ethernet links in this example constitute a bundle of communication links. Link Aggregation Control Protocol (LACP) is part of an IEEE specification (802.3ad) that allows several physical ports to be bundled together to form a single logical channel. LACP allows a switch to negotiate an automatic bundle by sending LACP packets to the adjacent node.
Bundled communication links are commercially available. For example bundled Ethernet links are available from Cisco Systems, Inc. of San Jose, Calif. as ETHERCHANNEL™ capabilities on Ethernet switches and routers. As another example, one of the two connected devices bundle the links into a group for redundancy using FLEXLINK. Bundled links are also available on routers for use with SONET for optical communication links as part of packet over SONET (POS) technology from Cisco Systems.
A load-balancing process is used on the sending network node of the pair connected by a bundle of communication links for the purpose of determining which communication link to use for sending one or more data packets to the receiving network node of the pair. A goal of load-balancing is to prevent one link from being heavily utilized to the point of congestion, while one or more other links in the bundle are under-utilized. Current balancing algorithms use a variety or methods to associate data packets with a specific port in a set of ports connected to the communication links in the bundle. Typically, information in a header portion of a data packet is used to derive a value that is associated with one port of the set, often using a hash function to make the association. The algorithm is designed to generate a value in a range of values that are associated with the full set of ports. Thus data packets directed to the receiving node are distributed over all communication links in the bundle by the load balancing process. Many load-balancing processes are designed so that all data packets in the same data flow are sent through the same port.
As more enterprises access wider metropolitan area networks for communication with remote sites, the enterprise procures service from a service provider for access to a MAN and WAN. The enterprise equipment is connected to service provider equipment at the edge of the service provider network. It is common for a service provider to provide a pair of so called edge nodes at the service provider edge so that service to the customer is not interrupted if one device fails.
Data packets are transferred across the provider network using virtual point-to-point links, so that traffic from one customer is not seen by other customers using the provider network. Such virtual point-to-point links appear to end users to be direct connections, but in fact may involve multiple intermediate network nodes. Virtual point-to-point links at layer 2 (data-link layer) also called pseudowires (PW) are available in a variety of protocols for a variety of different physical media. Such protocols include Multi-Protocol Label switching (MPLS), shared packet rings, and resilient packet ring (RPR) protocol for use over Ethernet media as well as optical media. Other protocols include MAC-in-MAC and Generic Routing Encapsulation (GRE) tunnels for Ethernet media. One or more pseudo wires are collected to form a virtual local area network (VLAN) or a virtual private network (VPN).
Virtual point-to-point links through the pair of devices on a provider edge provide redundancy in case of failure of either device using available processes, such as Spanning Tree or Back-Up PW. However, no previously known processes provide load balancing among data-links from one device to two or more other intermediate network nodes, such as the pair of provider edge nodes. Thus the capability to load-balance traffic from remote sites across the pair of provider edge nodes is not made available. As a result, one link or link-bundle between the customer device and one provider edge node may become congested while one or more links or link-bundles of a different provider edge node to the same customer network are under-utilized.
Based on the foregoing, there is a clear need for a load-balancing process for data-links distributed across the pair of provider edge nodes. In general, there is a need for a load-balancing process for data links to a given node when the data links are distributed across two or more different intermediate network nodes.